Jovian M.L Project (PUBG Finish Placement Prediction)¶
You are given over 65,000 games' worth of anonymized player data, split into training and testing sets, and asked to predict final placement from final in-game stats and initial player ratings.
What's the best strategy to win in PUBG? Should you sit in one spot and hide your way into victory, or do you need to be the top shot? Let's let the data do the talking!
Link to dataset: HERE¶
Data Description¶
Features¶
DBNOs - Number of enemy players knocked.
assists - Number of enemy players this player damaged that were killed by teammates.
boosts - Number of boost items used.
damageDealt - Total damage dealt. Note: Self inflicted damage is subtracted.
headshotKills - Number of enemy players killed with headshots.
heals - Number of healing items used.
Id - Player’s Id
killPlace - Ranking in match of number of enemy players killed.
killPoints - Kills-based external ranking of player. (Think of this as an Elo ranking where only kills matter.) If there is a value other than -1 in rankPoints, then any 0 in killPoints should be treated as a “None”.
killStreaks - Max number of enemy players killed in a short amount of time.
kills - Number of enemy players killed.
longestKill - Longest distance between player and player killed at time of death. This may be misleading, as downing a player and driving away may lead to a large longestKill stat.
matchDuration - Duration of match in seconds.
matchId - ID to identify match. There are no matches that are in both the training and testing set.
matchType - String identifying the game mode that the data comes from. The standard modes are “solo”, “duo”, “squad”, “solo-fpp”, “duo-fpp”, and “squad-fpp”; other modes are from events or custom matches.
rankPoints - Elo-like ranking of player. This ranking is inconsistent and is being deprecated in the API’s next version, so use with caution. Value of -1 takes place of “None”.
revives - Number of times this player revived teammates.
rideDistance - Total distance traveled in vehicles measured in meters.
roadKills - Number of kills while in a vehicle.
swimDistance - Total distance traveled by swimming measured in meters.
teamKills - Number of times this player killed a teammate.
vehicleDestroys - Number of vehicles destroyed.
walkDistance - Total distance traveled on foot measured in meters.
weaponsAcquired - Number of weapons picked up.
winPoints - Win-based external ranking of player. (Think of this as an Elo ranking where only winning matters.) If there is a value other than -1 in rankPoints, then any 0 in winPoints should be treated as a “None”.
groupId - ID to identify a group within a match. If the same group of players plays in different matches, they will have a different groupId each time.
numGroups - Number of groups we have data for in the match.
maxPlace - Worst placement we have data for in the match. This may not match with numGroups, as sometimes the data skips over placements.
Target¶
- winPlacePerc - The target of prediction. This is a percentile winning placement, where 1 corresponds to 1st place, and 0 corresponds to last place in the match. It is calculated off of maxPlace, not numGroups, so it is possible to have missing chunks in a match.
Since the target column is a continous value we can use Regression technique. But first we need to take a look at the data and perform EDA and other techniques as and when required.¶
In [1]:
# Required Imports
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
%matplotlib inline
warnings.filterwarnings("ignore")
pd.set_option('display.max_columns', 200)
sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (10, 6)
matplotlib.rcParams['figure.facecolor'] = '#00000000'
In [2]:
# Loading the datasets
train_df = pd.read_csv("train_V2.csv")
test_df = pd.read_csv("test_V2.csv")
print(f"Training data Shape: {train_df.shape}\nTest data shape: {test_df.shape}")
In [3]:
# First five rows of the train data
train_df.head()
Out[3]:
In [4]:
# First five rows of the test data
test_df.head()
Out[4]:
Approach Map¶
Exploratory Data Analysis
Outlier removal
Feature Engineering
Model selection, training & prediction
1. Exploratory Data Analysis¶
In [5]:
# Get dtypes of all columns
train_df.info()
We can observe that majority of the columns are either of type Int or Float.¶
In [6]:
# Statistical info about the numeric columns
train_df.describe()
Out[6]:
In [7]:
# NaN values count
train_df.isna().sum()
Out[7]:
Only one missing value so we can choose to drop this row.¶
In [8]:
# Dropping the only row with missing value
train_df.dropna(inplace=True)
test_df.dropna(inplace=True)
Since there are only 4 object type columns we can explore them first.¶
In [9]:
object_types = train_df.select_dtypes("object").columns.to_list()
object_types
Out[9]:
In [10]:
# Working with different IDs
for ids in object_types[:-1]:
print(f"Unique points in {ids} column are {train_df[ids].nunique()}")
Some Insights from the above operation:¶
Looks like ID column has only unique value present and that makes sense as each player is assigned a unique "PlayerID" in the game.
GroupID is almost the half of the PlayerID which may point to the fact that most players like to play group matches
In [11]:
# Lets look at the last "matchType" column
print(f"There are {train_df['matchType'].nunique()} categories in the Match type column")
In [12]:
# Value count of each category
train_df["matchType"].value_counts()
Out[12]:
In [13]:
# Visualization of the above data
sns.countplot(x="matchType", data=train_df)
plt.title("Match type Category-wise frequency")
plt.xticks(rotation=75)
plt.show()
As mentioned above looks like most of the players like to play squad matches. And we can also observe that the present 16 categories can be narrowed down to just 3 categories namely:¶
Squad
Duo
Solo
In [14]:
# Narrowing the present 16 categories to 3 categories
mapper = lambda x: 'solo' if ('solo' in x) else 'duo' if ('duo' in x) else 'squad'
train_df['matchType'] = train_df['matchType'].apply(mapper)
sns.countplot(train_df['matchType'])
plt.title('Count of different types of match')
plt.show()
In [15]:
# Applying the same to test set
test_df["matchType"] = test_df["matchType"].apply(mapper)
test_df.head()
Out[15]:
Exploring the numeric columns¶
In [16]:
# Let's visualize the correlation of the features with target
plt.figure(figsize=(15, 15))
correlation = train_df.corr()
sns.heatmap(correlation, annot=True, fmt="0.1f", cmap="summer")
plt.title("Correlation Matrix Heatmap")
plt.show()
From the above heatmap we can infer the following:¶
kills has a strong positive correlation with win %age.
boosts has a strong positive correlation with win %age.
weaponsAcquired has a nearly perfect correlation with win %age.
walkDistance has a perfect correlation with win %age.
heal has a strong positive correlation with win %age.
damageDealt has a strong positive correlation with win %age.
Let's analyze each one of them.
Analyzing the Kills¶
In [17]:
print("The average kill by a player are:", train_df["kills"].mean())
print(f"The minimum kills by a player are: {train_df['kills'].min()} and maximum kills are: {train_df['kills'].max()}")
In [18]:
# Unique kills value
train_df["kills"].unique()
Out[18]:
In [19]:
train_df["kills"].value_counts()
Out[19]:
We can see that there are quite a lot of values of no. of kills. To visualize the number of kills we can map the no kills with a value greater than 6 to something such "Greater than 7" since most of the kills are in the range of 0 to 7.¶
In [20]:
kills = train_df["kills"].copy()
kills = kills.apply(lambda x: "Greater than 7" if x > 6 else x)
kills.value_counts()
Out[20]:
In [21]:
# Visualizing the kills count
sns.countplot(kills.astype(str).sort_values())
plt.title("No of kills in the game")
plt.xticks(rotation=45)
plt.show()
If we take no of kills as the main factor in winning the game then it seems that most of the players are quite average at playing the game.¶
In [22]:
sns.scatterplot(x="winPlacePerc", y="kills", data=train_df, alpha=0.7)
plt.show()
We can look at the longestKill column as well. Typically, long kills are made using Snipers that too within a range of 800 meters. So we can drop those players having a longKill value of greater than 800 meters as they might be playing with some hacks and mods¶
In [23]:
# Players who killed at a range greater than 800 meters
longest = train_df[train_df["longestKill"] > 800]
longest
Out[23]:
In [24]:
# Visualizing long shots by hackers
sns.histplot(longest["longestKill"], bins=10)
plt.title("Count of players killing at a range >800 meters")
plt.show()
From the above visualization it can be concluded that these players are hackers. So we can choose to drop them from our dataframe¶
In [25]:
# Boxplot of no of kills.
kills = train_df.copy()
kills['killsCategories'] = pd.cut(kills['kills'], [-1, 0, 2, 5, 10, 60], labels=[
'0_kills','1-2_kills', '3-5_kills', '6-10_kills', '10+_kills'])
plt.figure(figsize=(15,8))
sns.boxplot(x="killsCategories", y="winPlacePerc", data=kills)
plt.show()
Though there are some outliers in the kills column but it seems that kills is correlated with win %age.¶
Analyzing boosts column¶
In [26]:
print("Average boost items used by players:", train_df["boosts"].mean())
print(f"Minimum no. of boosts item used are {train_df['boosts'].min()} and maximum are {train_df['boosts'].max()}")
In [27]:
train_df["boosts"].value_counts()
Out[27]:
In [28]:
# Visualizing boosts
sns.countplot(train_df["boosts"].sort_values(ascending=False))
plt.show()
The maximum no. of boost items used by players is 0, which means that either the players died too early in the match or if they win without using any boosts then most probably they are hackers.¶
In [29]:
sns.scatterplot(x="winPlacePerc", y="boosts", data=train_df, color="magenta", alpha=0.7)
plt.show()
Analyzing weaponsAcquired¶
In [30]:
print("The average number of weapons acquired by players are:", train_df["weaponsAcquired"].mean())
print(f"Minimum no. of weapons acquired are {train_df['weaponsAcquired'].min()} and maximum are {train_df['weaponsAcquired'].max()}")
The maximum value of weapons acquired is 236 which is basically impossible in a single match. The min. number is 0 which tells us that the players died before acquiring any weapons.(i.e at the start of the match)¶
In [31]:
sns.scatterplot(x="winPlacePerc", y="weaponsAcquired", data=train_df, color="orange", alpha=0.7)
plt.show()
Since each player start with no weapons and tries to acquire better weapons throughout the match, it can be concluded that the more the weapons acquired the more is the probability of winning. (Although there are some outliers needed to be taken care of).¶
In [32]:
# No weapons acquired but still won the game
no_weapons_won = train_df[(train_df["weaponsAcquired"] == 0) & (train_df["winPlacePerc"] == 1)]
no_weapons_won
Out[32]:
We can observe that there are around 200 players who didn't even use a weapon but still managed to win the game. Either they were sitting at a corner through out the game or they are fraudsters.¶
Analyzing walk distance¶
In [33]:
print(f"The average walk distance is {train_df['walkDistance'].mean()} meters.")
In [34]:
train_df["walkDistance"].value_counts()
Out[34]:
About 99600 players walked 0 meters which means that they either died at the very start or they got disconnected from the game.¶
In [35]:
# Visualizing walk distance
data = train_df[train_df['walkDistance'] < train_df['walkDistance'].quantile(0.99)]
plt.figure(figsize=(15,7))
plt.title("Walking Distance Distribution")
sns.distplot(data['walkDistance'])
plt.show()
In [36]:
sns.scatterplot(x="winPlacePerc", y="walkDistance", data=train_df, color="green", alpha=0.7)
plt.show()
We can observe that we also have swimDistance and rideDistance. Combining all these 3 distances we can derive a new column. Theoretically, if a player has travelled a lot of distance then there are high chances that the player may end up winning the match.¶
In [37]:
train_df["totalDistance"] = train_df["walkDistance"] + train_df["rideDistance"] + train_df["swimDistance"]
train_df["totalDistance"].head()
Out[37]:
In [38]:
test_df["totalDistance"] = test_df["walkDistance"] + test_df["rideDistance"] + test_df["swimDistance"]
test_df["totalDistance"].head()
Out[38]:
In [39]:
plt.figure(figsize=(15, 8))
sns.distplot(train_df["totalDistance"])
plt.show()
To win a game in a game like PUBG, one has to keep moving from one place to another and stay inside the safe zone. If player managed to win a game after covering covering a very minimal distance, then it can be concluded that either they were camping or they are hackers.¶
In [40]:
minimal_movement = train_df[(train_df["totalDistance"] <= 350) & (train_df["winPlacePerc"]) == 1]
minimal_movement
Out[40]:
In [41]:
# Visualizing above data
sns.histplot(minimal_movement["totalDistance"], bins=10, color="purple")
plt.show()
As we can observe from the above graph that there are quite a lot of players who covered minimal distance but still managed to win the match.¶
In [42]:
sns.scatterplot(x="winPlacePerc", y="totalDistance", data=train_df, color="indigo", alpha=0.7)
plt.show()
Analyzing Heals vs Boosts¶
In [43]:
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(15, 10))
sns.countplot(x="boosts", data=train_df[train_df["boosts"] < train_df["boosts"].quantile(0.99)], ax=axes[0])
sns.countplot(x="heals", data=train_df[train_df["heals"] < train_df["heals"].quantile(0.99)], ax=axes[1])
plt.show()
From the above visualization it can be concluded that most of the players use heals instead of boosts.¶
In [44]:
# Visualizing effect of boosts and heals on Win %age
data = train_df[train_df['heals'] < train_df['heals'].quantile(0.99)]
data = data[data['boosts'] < data['boosts'].quantile(0.99)]
plt.figure(figsize=(15, 8))
sns.pointplot(x='heals', y='winPlacePerc', data=data,
color='purple', alpha=0.8)
sns.pointplot(x='boosts', y='winPlacePerc', data=data,
color='orange', alpha=0.8)
plt.title("Heals vs Boosts", fontdict={"size": 25, "weight": "bold"})
plt.xlabel("No of heals/boosts", fontdict={"size": 25})
plt.ylabel("Win %age", fontdict={"size": 25})
plt.text(0.15,0.85,'Heals', color='purple', fontsize = 20)
plt.text(0.15,0.80,'Boosts', color='orange', fontsize = 20)
plt.grid()
plt.tight_layout()
plt.show()
From the above plot it can be stated that more the amount of heals and boosts a player has, the more is the probability of winning the match.¶
Analyzing Damage dealt¶
In [45]:
print("The average damage dealt by players is:", train_df["damageDealt"].mean())
print(f"The minimum damage dealt is {train_df['damageDealt'].min()} and maximum is {train_df['damageDealt'].max()}")
In [46]:
train_df["damageDealt"].value_counts()
Out[46]:
In [47]:
no_dmg_players = train_df[(train_df["damageDealt"] == 0) & (train_df["winPlacePerc"] == 1)]
no_dmg_players
Out[47]:
In [48]:
print("Average kill points of above data:", no_dmg_players["killPoints"].mean())
We can see that the above players have dealt 0 damage but they still some of them have killPoints (even without kiling anyone?). And the average kill points comes out to be 430.¶
In [49]:
# Visualizing the above data
sns.distplot(no_dmg_players["killPoints"])
plt.show()
2. Outlier Removal¶
From the above data analysis, we can drop out the shady/highly unlikely data points.¶
Shady/Unlikely data points¶
Longest kills of >800 meters.
Winning without boosts and heals.
Travelling long distances by foot without killing anyone.
Acquiring large number of weapons and killing more than 35 players. (Sign of hackers)
Gaining high number of kills without covering much distance/ without movements.
Gaining high no. of road kills.
Travelling a total distance of 0 meters and still winning the game.
In [50]:
to_drop = train_df[train_df["longestKill"] > 800]
train_df.drop(to_drop.index, inplace=True)
print(f"Dropped {len(to_drop)} entries of Longest kills >800 meters.")
In [51]:
train_df["heals"].mean(), train_df["boosts"].mean()
Out[51]:
In [52]:
to_drop = train_df[(train_df["heals"] < 2) & (train_df["boosts"] < 2) & (train_df["winPlacePerc"] == 1)]
train_df.drop(to_drop.index, inplace=True)
print(f"Dropped {len(to_drop)} fraudsters winning with negligible boosts and heals.")
In [53]:
train_df["walkDistance"].describe()
Out[53]:
In [54]:
to_drop = train_df[(train_df["walkDistance"] > 2000) & (train_df["winPlacePerc"] == 1)]
train_df.drop(to_drop.index, inplace=True)
print(f"Dropped {len(to_drop)} players walking long distances without encountering any players.")
In [55]:
train_df[(train_df["weaponsAcquired"] > 30) & (train_df["kills"] > 30)].describe()
Out[55]:
In [56]:
to_drop = train_df[(train_df["weaponsAcquired"] > 30) & (train_df["kills"] > 30)]
train_df.drop(to_drop.index, inplace=True)
print(f"Dropped {len(to_drop)} players with highly unlikely weapons possession and kills.")
In [57]:
# Drop travelling outliers/anomalities
to_drop_walk = train_df[(train_df['walkDistance'] == 0) & (train_df['rideDistance'] > 0) & (train_df['kills'] > 0)]
to_drop_ride = train_df[(train_df['rideDistance'] >= 25000) & (train_df['kills'] == 0)]
to_drop_swim = train_df[(train_df["swimDistance"] >= 1500) & (train_df["kills"] == 0)]
to_drop_total = train_df[(train_df["totalDistance"] == 0) & (train_df["kills"] > 0)]
In [58]:
train_df.drop(to_drop_walk.index, inplace=True)
train_df.drop(to_drop_ride.index, inplace=True)
train_df.drop(to_drop_swim.index, inplace=True)
train_df.drop(to_drop_total.index, inplace=True)
print(f"Dropped {len(to_drop_walk)} players with no walk distance but still managed to get kills.")
print(f"Dropped {len(to_drop_ride)} players riding a distance >=25000 meters without getting any kills.")
print(f"Dropped {len(to_drop_swim)} players swimming long distances without encountering any players.")
print(f"Dropped {len(to_drop_total)} players who stayed still or were away from keyboard/disconnected.")
We have analyzed our data and the anomalities/outliers have been handled.¶
In [59]:
# Let's look at the shape of final data
train_df.shape
Out[59]:
In [60]:
# Separating features and Target
X = train_df.drop("winPlacePerc", axis=1)
y = train_df["winPlacePerc"]
In [61]:
print("winPlacePerc" in X.columns)
In [62]:
y.head()
Out[62]:
3. Feature Engineering¶
Let's take a look at the correlation matrix of our features data.¶
In [63]:
corr = X.corr().abs()
corr
Out[63]:
In [64]:
# Visualizing the above matrix
plt.figure(figsize=(15, 10))
sns.heatmap(corr, annot=True, fmt="0.1f", cmap="plasma")
plt.show()
We can notice that we have few features that are correlated with others. This may affect our model predictions, so we can drop these¶
Handling highly correlated features¶
In [65]:
# Getting the upper triangluar matrix of correlated data
upper_tri = corr.where(np.triu(np.ones(corr.shape),k=1).astype(np.bool))
upper_tri
Out[65]:
We got the upper triangluar matrix, now we only need to select the features that are highly correlated with each other¶
In [66]:
to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > 0.8)] # Keeping the threshold value as 0.6
to_drop
Out[66]:
In [67]:
X.drop(X[to_drop + ["Id", "groupId", "matchId"]], axis=1, inplace=True)
X.head()
Out[67]:
We can also drop the Id columns as they are unique and won't contribute that much to our model training /predictions¶
In [68]:
test_df.drop(to_drop + ["Id", "groupId", "matchId"], axis=1, inplace=True)
Splitting the data into train and validation test.¶
In [69]:
from sklearn.model_selection import train_test_split
train, val = train_test_split(pd.concat([X, y], axis=1), test_size=0.2, random_state=42)
print("Size of training set:", train.shape)
print("Size of validation set:", val.shape)
In [70]:
train_inputs, train_target = train.drop("winPlacePerc", axis=1), train["winPlacePerc"]
val_inputs, val_target = val.drop("winPlacePerc", axis=1), val["winPlacePerc"]
In [71]:
numeric_cols = train_inputs.select_dtypes(include=np.number).columns.tolist()
categorical_cols = train_inputs.select_dtypes('object').columns.tolist()
Scaling numeric values¶
In [72]:
# Scaling numeric values using MinMaxScaler
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
scaler.fit(train_df[numeric_cols])
Out[72]:
In [73]:
train_inputs[numeric_cols] = scaler.transform(train_inputs[numeric_cols])
val_inputs[numeric_cols] = scaler.transform(val_inputs[numeric_cols])
test_df[numeric_cols] = scaler.transform(test_df[numeric_cols])
Handling categorical columns¶
In [74]:
print(f"There are {train_inputs['matchType'].nunique()} different Match types in the dataset.")
In [75]:
# We can one-hot encode these
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown="ignore")
encoder.fit(train_df[["matchType"]])
Out[75]:
In [76]:
encoded_cols = list(encoder.get_feature_names(["matchType"]))
encoded_cols
Out[76]:
In [77]:
train_inputs[encoded_cols] = encoder.transform(train_inputs[["matchType"]]).toarray()
val_inputs[encoded_cols] = encoder.transform(val_inputs[["matchType"]]).toarray()
test_df[encoded_cols] = encoder.transform(test_df[["matchType"]]).toarray()
In [78]:
train_inputs.drop("matchType", axis=1, inplace=True)
val_inputs.drop("matchType", axis=1, inplace=True)
test_df.drop("matchType", axis=1, inplace=True)
In [79]:
train_inputs.head()
Out[79]:
In [80]:
train_target.head()
Out[80]:
4. Model selection, training & prediction¶
Since we need to predict a continous value, we can consider using a regression model.¶
Model to consider¶
Linear Regressor
DecisionTree Regressor
Random Forest Regressor
Support Vector Regressor
4.1 Linear Regressor¶
In [81]:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
In [82]:
linear_model = LinearRegression(n_jobs=-1, fit_intercept=True)
linear_model.fit(train_inputs, train_target) # The LinearRegression model does not have mucb hyperparameters to tune.
Out[82]:
In [83]:
linear_model.coef_
Out[83]:
In [84]:
linear_model.score(train_inputs, train_target)
Out[84]:
Looks like our Linear model was able to learn the patterns in the data and got a respectable score of 0.80¶
In [85]:
linear_train_preds = linear_model.predict(train_inputs)
In [86]:
# Let's check how our model performs on validation set
linear_model.score(val_inputs, val_target)
Out[86]:
It seems that our model has recognized the patterns well on the data it has never seen before¶
In [87]:
linear_val_preds = linear_model.predict(val_inputs)
linear_val_preds
Out[87]:
In [88]:
mean_squared_error(val_target, linear_val_preds, squared=False)
Out[88]:
The RMSE is also acceptable.¶
In [89]:
# Time to Visualize our linear model but since out dataset is very large we can use only few datapoints
fig, axes = plt.subplots(1, 2)
sns.regplot(train_target[:250], linear_train_preds[:250], color="red",
scatter_kws={"color": "violet", "alpha": 0.6}, ax=axes[0],
label="Train Regression")
sns.regplot(val_target[:250], linear_val_preds[:250], color="red",
scatter_kws={"color": "seagreen", "alpha": 0.7}, ax=axes[1],
label="Validation Regression")
fig.legend(loc=(0.07, 0.85))
plt.show()
Our linear model was able to fit a respectable regression line.¶
Since we will be visualizing the predictions lets define a separate function for that
In [90]:
def visualize_error_predictions(train_preds, val_preds):
"""Plots the Train vs Validation errors and
Actual vs Predicted datapoints"""
fig, axes = plt.subplots(1, 3, figsize=(25, 10))
train_error = (train_target - train_preds)[:500]
val_error = (val_target - val_preds)[:500]
axes[0].scatter(train_error, range(500), color="green",
alpha=0.7, label="Train", marker="*")
axes[0].scatter(val_error, range(500), alpha=0.7, color="royalblue", label="Validation")
axes[1].plot(range(100), train_target[:100], alpha=0.65, color="purple", label="Train Target")
axes[1].plot(range(100), train_preds[:100], alpha=0.65, color="red", label="Predicted")
axes[2].plot(range(100), val_target[:100], alpha=0.65, label="Val. Target", color="cyan")
axes[2].plot(range(100), val_preds[:100], alpha=0.65,
label="Predicted", color="green")
axes[0].set_title("Train Error vs Validation Error", fontdict={"size":23, "weight": "bold"})
axes[0].set_xlabel("Error", fontdict={"size":20})
axes[0].set_ylabel("Data Points", fontdict={"size":20})
axes[0].legend()
axes[1].set_title("Train Targets vs Predicted Targets", fontdict={"size":23, "weight": "bold"})
axes[1].set_xlabel("Data Points", fontdict={"size":20})
axes[1].set_ylabel("Predictions", fontdict={"size":20})
axes[1].legend()
axes[2].set_title("Validation Targets vs Predicted Targets", fontdict={"size":23, "weight": "bold"})
axes[2].set_xlabel("Data Points", fontdict={"size":20})
axes[2].set_ylabel("Predictions", fontdict={"size":20})
axes[2].legend()
plt.show()
In [91]:
visualize_error_predictions(linear_train_preds, linear_val_preds)
From the above visualiztion we can conclude that our Linear model performed identical on both the training as well as the validation test¶
Linear Model Summary:¶
Performed identically well on both Training and Validation data
Training accuracy: 0.8029, Validation Accuracy: 0.8025
RMSE Value of Validation set: 0.1327
4.2 Decision Tree Regressor¶
In [92]:
from sklearn.tree import DecisionTreeRegressor
from sklearn.tree import plot_tree
In [93]:
# Training decision tree with default hyper params
tree_regressor = DecisionTreeRegressor()
tree_regressor.fit(train_inputs, train_target)
Out[93]:
In [94]:
tree_train_preds = tree_regressor.predict(train_inputs)
tree_train_preds
Out[94]:
Let's check the model score on training set¶
In [95]:
tree_regressor.score(train_inputs, train_target)
Out[95]:
And our Decision Tree model is close to being 100 %. accurate but is it the case since Decision Trees are famous for overfitting the data and giving high training set scores.¶
In [96]:
# Checking score on validation set
tree_regressor.score(val_inputs, val_target)
Out[96]:
So looks like our model has definitely overfit the training data as it only scores around 83% on validation set.¶
Let's visualize both the training and validation predictions
In [97]:
tree_val_preds = tree_regressor.predict(val_inputs)
tree_val_preds
Out[97]:
In [98]:
visualize_error_predictions(tree_train_preds, tree_val_preds)
The straight line means that our model is making near perfect predictions on the training set (the error is almost 0) but fails to do so on validation set (represented by scattered dots). Also it may look like that there is just a single line plotted but actually the predicted line has overlapped on the actual line as out DT has overfitted out data.¶
In [99]:
# Visualize the structure of our tree
plt.figure(figsize=(15, 6))
plot_tree(tree_regressor, feature_names=train_inputs.columns, max_depth=2, filled=True)
plt.show()
Let's take a look at the importance value that our DT has assigned to each feature¶
In [100]:
importance_df = pd.DataFrame(
{
"Feature": train_inputs.columns,
"Importance": tree_regressor.feature_importances_
}
).sort_values("Importance", ascending=False)
importance_df
Out[100]:
In [101]:
# Visualizing the feature importance
plt.figure(figsize=(15, 9))
sns.barplot(x="Importance", y="Feature", data=importance_df, orient="h")
plt.title("Importance value assigned by Decision Tree", fontdict={"size":20, "weight": "bold"})
plt.show()
Let's perform some HyperParameters tuning so as to improve our model performance.¶
Going through the official scikit-learn docs on Decision Tree Regressor, the following hyperparamters can be tuned
max_depth
max_leaf_nodes
splitter
max_features
Manually searching for best hyper-params can be a tedious task. So we can use a technique called GridSearch cross-validation to find the best hyper-params value.¶
In [102]:
from sklearn.model_selection import GridSearchCV
In [106]:
params = {
"max_depth": np.random.randint(3, 11, size=5),
"max_leaf_nodes": np.random.randint(0, 15, size=5),
"splitter": ["best", "random"],
"max_features": ["auto","log2","sqrt",None]
}
tuning_model = GridSearchCV(tree_regressor, param_grid=params,
scoring='neg_mean_squared_error', cv=3,
verbose=3)
Our tuning model has been initialized. We need to train it on the training data so that it can calculate the best hyper-params for our DT model.¶
In [107]:
%%time
tuning_model.fit(train_inputs, train_target)
Out[107]:
The Grid Search cross-validation took about 45 mins to complete, but this time may vary from machine to machine and size of dataset. Now we can take a look at the best hyper-params that the CV has retained for our DT model.¶
In [108]:
# Best parameters
best_hyper_params = tuning_model.best_params_
best_hyper_params
Out[108]:
In [109]:
# Best model score
tuning_model.best_score_
Out[109]:
Now we can train a new DT model with the best hyper-params¶
In [110]:
hyper_tuned_dt = DecisionTreeRegressor(**best_hyper_params)
hyper_tuned_dt.fit(train_inputs, train_target)
Out[110]:
In [111]:
hyper_model_train_preds = hyper_tuned_dt.predict(train_inputs)
hyper_model_train_preds
Out[111]:
In [112]:
# Checking the score of hyper-tuned model on training set
hyper_tuned_dt.score(train_inputs, train_target)
Out[112]:
We can see that the training accuracy dropped significantly, from almost 100% it dropped down to about 82%. Let's check if there is any change in the validation set score or not.¶
In [113]:
# Checking the score of hyper-tuned model on training set
hyper_model_val_preds = hyper_tuned_dt.predict(val_inputs)
hyper_tuned_dt.score(val_inputs, val_target)
Out[113]:
In [114]:
importance_df_hyper = pd.DataFrame(
{
"Feature": train_inputs.columns,
"Importance": hyper_tuned_dt.feature_importances_
}
).sort_values("Importance", ascending=False)
importance_df_hyper
Out[114]:
Walk distance still has the highest importance but order of some features has been changed. A plot will help us understand it in a better way¶
In [115]:
fig, axes = plt.subplots(1, 2)
sns.barplot(x="Importance", y="Feature", data=importance_df.head(), orient="h", ax=axes[0])
sns.barplot(x="Importance", y="Feature", data=importance_df_hyper.head(), orient="h", ax=axes[1])
axes[0].set_title("Without Hyper-Param Tuning")
axes[1].set_title("With Hyper-Param Tuning")
fig.tight_layout()
plt.show()
So both walkDistance and killPlace remained highly valued features .¶
In [116]:
mean_squared_error(val_target, hyper_model_val_preds)
Out[116]:
Visulizing error of our Hyper Tuned model¶
In [117]:
visualize_error_predictions(hyper_model_train_preds, hyper_model_val_preds)
In [118]:
# Visualizing structure of our hyper tuned tree
plt.figure(figsize=(20, 8))
plot_tree(hyper_tuned_dt, max_depth=3, filled=True,
feature_names=train_inputs.columns, fontsize=12)
plt.show()
DT Regressor Model Summary:¶
It is more prone to Overfitting.
Performed identically well on both Training and Validation data after HyperParam tuning
Training accuracy: 0.9999, Validation Accuracy: 0.8319 (Without Hyper Param tuning)
Training accuracy: 0.8201, Validation Accuracy: 0.8193 (With Hyper Param tuning)
RMSE Value of Validation set: 0.0161
4.3 Random Forest Regressor¶
In [119]:
from sklearn.ensemble import RandomForestRegressor
In [120]:
# Training the random forest with default hyper-params
rf_regressor = RandomForestRegressor(n_estimators=10, n_jobs=-1)
rf_regressor.fit(train_inputs, train_target)
Out[120]:
Note: n_estimators has been set to 10 as it took more than 45 mins to train the model with default value set to 100.
Model score on training set¶
In [121]:
rf_regressor.score(train_inputs, train_target)
Out[121]:
In [122]:
rf_regressor.score(val_inputs, val_target)
Out[122]:
That's interesting! Our RF Model scores around 91% on the data that it has never encountered before. So it seems that our model has not completely overfit the data. However, with some hyper-param tuning we can further improve our model performance.¶
Visualizing feature importance (without hyper param tuning)¶
In [123]:
# Visualizing error in predictions (without hyper param tuning)
importance_df_hyper = pd.DataFrame(
{
"Feature": train_inputs.columns,
"Importance": rf_regressor.feature_importances_
}
).sort_values("Importance", ascending=False)
importance_df_hyper
Out[123]:
In [124]:
plt.figure(figsize=(15, 9))
sns.barplot(x="Importance", y="Feature", data=importance_df, orient="h")
plt.title("Importance value assigned by RF Regressor", fontdict={"size":20, "weight": "bold"})
plt.show()
In [125]:
rf_train_preds = rf_regressor.predict(train_inputs)
rf_val_preds = rf_regressor.predict(val_inputs)
In [126]:
visualize_error_predictions(rf_train_preds, rf_val_preds)
As we can see that the error on the training set is close to 0 where as on the validation set it is much more scattered through out the plot but majority of the data points are tending to 0.¶
Let's perform some HyperParameters tuning so as to improve our model performance.¶
Going through the official scikit-learn docs on Random Tree Regressor, the following hyperparamters can be tuned
n_estimators (we will set it to 10 so as to save some time but that does not mean that this is the optimal value)
max_depth
max_leaf_nodes
max_features
We can use the previous optimal hyper-params (excluding splitters) since GridSearch is computationally expensive and for RandomForest it may take more time.¶
In [127]:
best_hyper_params_rf = {key: best_hyper_params[key] for key in best_hyper_params if key != "splitter"}
best_hyper_params_rf
Out[127]:
In [128]:
hyper_tuned_rf = RandomForestRegressor(n_estimators=10, **best_hyper_params_rf)
hyper_tuned_rf.fit(train_inputs, train_target)
Out[128]:
In [129]:
hyper_rf_train_preds = hyper_tuned_rf.predict(train_inputs)
hyper_rf_val_preds = hyper_tuned_rf.predict(val_inputs)
It's time to check the score of our hyper param tuned model on both training as well as validation set.¶
In [130]:
hyper_tuned_rf.score(train_inputs, train_target)
Out[130]:
In [131]:
hyper_tuned_rf.score(val_inputs, val_target)
Out[131]:
In [132]:
mean_squared_error(val_target, hyper_rf_val_preds)
Out[132]:
In [133]:
visualize_error_predictions(hyper_rf_train_preds, hyper_rf_val_preds)
Let's check if there is any difference in feature importance after hyper param tuning.¶
In [134]:
importance_df_hyper = pd.DataFrame(
{
"Feature": train_inputs.columns,
"Importance": hyper_tuned_rf.feature_importances_
}
).sort_values("Importance", ascending=False)
importance_df_hyper
Out[134]:
In [135]:
fig, axes = plt.subplots(1, 2)
sns.barplot(x="Importance", y="Feature", data=importance_df.head(), orient="h", ax=axes[0])
sns.barplot(x="Importance", y="Feature", data=importance_df_hyper.head(), orient="h", ax=axes[1])
axes[0].set_title("Without Hyper-Param Tuning")
axes[1].set_title("With Hyper-Param Tuning")
fig.tight_layout()
plt.show()
Yes there is some change in the feature importance that can be noticed.¶
RF Regressor Model Summary:¶
It is too prone to Overfitting but still manages to generalize the data better than DTs without hyper param tuning.
Performed identically well on both Training and Validation data after HyperParam tuning
Takes much more time to train.
Training accuracy: 0.9837, Validation Accuracy: 0.9085 (Without Hyper Param tuning)
Training accuracy: 0.8273, Validation Accuracy: 0.8266 (With Hyper Param tuning)
RMSE Value of Validation set: 0.0154
Conclusion¶
The following conclusion can be made from the above analysis¶
walkDistance and killPoints remained the dominating features and largely impacted the predictions.
All the 3 models performed well.
In terms of reliability, RandomForest is the way to go.
LinearRegression was faster than both DT and RF.
RMSE of LinearRegression was much more than that of DT or RF.
Hyperparameters tuning greatly reduced overfitting in DT and RF model.
More features may lead to better accuracy score and predictions.
In [137]:
model_stats = {
"Name": ["Linear Regression", "DecisionTree Regressor", "RandomForest Regressor"],
"Accuracy on Training set (without Hyperparam tuning)": [0.8029, 0.9999, 0.9837],
"Accuracy on Validation set (without Hyperparam tuning)": [0.8025, 0.8319, 0.9085],
"Accuracy on Training set (with Hyperparam tuning)": [np.nan, 0.8233, 0.8303],
"Accuracy on Validation set (with Hyperparam tuning)": [np.nan, 0.8273, 0.8266],
"RMSE (of validation set)": [0.1327, 0.0158, 0.0154]
}
model_stats_df = pd.DataFrame(model_stats)
model_stats_df
Out[137]:
Before winding up, we can save our model as well as other important objects so that we can use them later without performing all of the preprocessing and training again.¶
In [138]:
import joblib
In [139]:
objects_to_save = {
"linear_model": linear_model,
"tree_model": hyper_tuned_dt,
"rf_model": hyper_tuned_rf,
"input_cols": X.columns.tolist(),
"target_cols": y,
"numeric_cols": X.select_dtypes(np.number).columns.tolist(),
"categorical_cols": ["matchType"],
"scaler": scaler,
"encoder": encoder
}
In [140]:
joblib.dump(objects_to_save, "pubg_predictions.joblib")
Out[140]:
And that's it! We have successfully handled a real world M.L problem.¶
References and Links¶
In [ ]: